Self-Supervised Visual Terrain Classification From Unsupervised Acoustic Feature Learning

Mobile robots operating in unknown urban environments encounter a wide range of complex terrains to which they must adapt their planned trajectory for safe and efficient navigation. Most existing approaches utilize supervised learning classify from either an exteroceptive or proprioceptive sensor modality. However, this requires tremendous amount manual labeling effort each newly encountered terrain as well variations caused by changing environmental conditions. In article, we propose novel classification framework leveraging unsupervised classifier that learns vehicle-terrain interaction sounds self-supervise pixelwise semantic segmentation images. To end, first learn discriminative embedding space triplets audio clips formed using visual features the corresponding patches cluster resulting embeddings. We subsequently use these clusters label projecting traversed tracks robot into camera Finally, sparsely labeled images train our network weakly manner. present extensive quantitative qualitative results demonstrate exceeds state-of-the-art among methods self-supervised model achieves comparable performance with manually data.